Multi-vehicle high integrity perception

ABSTRACT

The illustrative embodiments provide a method for processing sensor data and controlling the movement of a vehicle. In one illustrative embodiment, a vehicle having a plurality of sensors attempts to receive sensor data. In response to an inability of the vehicle to obtain needed sensor data, collected sensor data is requested from a plurality of other vehicles to form alternate sensor data. The alternate sensor data is received and the vehicle is controlled using the alternate sensor data. In another illustrative embodiment, a request is received at a first vehicle for sensor data from a different vehicle. Sensor data is collected from a plurality of sensors at the first vehicle. The sensor data is then sent to the different vehicle.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to commonly assigned and co-pending U.S.patent application Ser. No. ______ (Attorney Docket No. 18152-US)entitled “Leader-Follower Semi-Autonomous Vehicle with Operator onSide”; U.S. patent application Ser. No. ______ (Attorney Docket No.18563-US) entitled “Leader-Follower Fully-Autonomous Vehicle withOperator on Side”; U.S. patent application Ser. No. ______ (AttorneyDocket No. 18479-US) entitled “High Integrity Perception for MachineLocalization and Safeguarding”; U.S. patent application Ser. No. ______(Attorney Docket No. 18678-US) entitled “Distributed Knowledge Base ForVehicular Localization And Work-Site Management”; U.S. patentapplication Ser. No. ______ (Attorney Docket No. 18478-US) entitled“Distributed Knowledge Base Method For Vehicular Localization AndWork-Site Management”; U.S. patent application Ser. No. ______ (AttorneyDocket No. 18679-US) entitled “Distributed Knowledge Base Program ForVehicular Localization and Work-Site Management”; U.S. patentapplication Ser. No. ______ (Attorney Docket No. 18680-US) entitled“Vehicle With High Integrity Perception System”; and U.S. patentapplication Ser. No. ______ (Attorney Docket No. 18682-US) entitled“High Integrity Perception Program” all of which are hereby incorporatedby reference.

FIELD OF THE INVENTION

The present disclosure relates generally to systems and methods forvehicle navigation and more particularly systems and methods for highintegrity perception for controlling operation of a vehicle. As anexample, embodiments of this invention provide a method and systemutilizing a versatile robotic control module for localization andnavigation of a vehicle.

BACKGROUND OF THE INVENTION

An increasing trend towards developing automated or semi-automatedequipment is present in today's work environment. In some situationswith the trend, this equipment is completely different from theoperator-controlled equipment that is being replaced, and does not allowfor any situations in which an operator can be present or take overoperation of the vehicle. Such unmanned equipment can be unreliable dueto the complexity of systems involved, the current status ofcomputerized control, and uncertainty in various operating environments.As a result, semi-automated equipment is more commonly used. This typeof equipment is similar to previous operator-controlled equipment, butincorporates one or more operations that are automated rather thanoperator-controlled. This semi-automated equipment allows for humansupervision and allows the operator to take control when necessary.

SUMMARY

The illustrative embodiments provide a method for processing sensor dataand controlling the movement of a vehicle. In one illustrativeembodiment, a vehicle having a plurality of sensors attempts to receivesensor data. In response to an inability of the vehicle to obtain neededsensor data, collected sensor data is requested from a plurality ofother vehicles to form alternate sensor data. The alternate sensor datais received and the vehicle is controlled using the alternate sensordata. In another illustrative embodiment, a request is received at afirst vehicle for sensor data from a different vehicle. Sensor data iscollected from a plurality of sensors at the first vehicle. The sensordata is then sent to the different vehicle.

The features, functions, and advantages can be achieved independently invarious embodiments of the present invention or may be combined in yetother embodiments in which further details can be seen with reference tothe following description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed characteristic of the invention are setforth in the appended claims. The invention itself, however, as well asa preferred mode of use, further objectives and advantages thereof, willbest be understood by reference to the following detailed description ofan illustrative embodiment when read in conjunction with theaccompanying drawings, wherein:

FIG. 1 is a block diagram of multiple vehicles operating in a networkenvironment in accordance with an illustrative embodiment;

FIGS. 2A and 2B are a block diagram illustrating vehicle perception usedto adjust navigation in accordance with an illustrative embodiment;

FIG. 3 is a block diagram of components used to control a vehicle inaccordance with an illustrative embodiment;

FIG. 4 is a block diagram of a data processing system in accordance withan illustrative embodiment;

FIG. 5 is a block diagram of a sensor system in accordance with anillustrative embodiment;

FIG. 6 is a block diagram of functional software components that may beimplemented in a machine controller in accordance with an illustrativeembodiment;

FIG. 7 is a block diagram of a knowledge base in accordance with anillustrative embodiment;

FIG. 8 is a block diagram of a sensor selection table in a knowledgebase used to select sensors for use in planning paths and obstacleavoidance in accordance with an illustrative embodiment;

FIG. 9 is a flowchart illustrating a process for sensor selection inaccordance with an illustrative embodiment;

FIG. 10 is a flowchart illustrating a process for prioritizing sensordata in accordance with an illustrative embodiment;

FIG. 11 is a flowchart illustrating a process for object classificationin accordance with an illustrative embodiment;

FIG. 12 is a flowchart illustrating a process for processing objectanomalies in accordance with an illustrative embodiment;

FIG. 13 is a flowchart illustrating a process for generating a thematicmap in accordance with an illustrative embodiment;

FIG. 14 is a flowchart illustrating a process for monitoring sensorintegrity in accordance with an illustrative embodiment;

FIG. 15 is a flowchart illustrating a process for requestinglocalization information from another vehicle in accordance with anillustrative embodiment;

FIG. 16 is a flowchart illustrating a process for transmittinglocalization information to another vehicle in accordance with anillustrative embodiment;

FIG. 17 is a flowchart illustrating a process for selecting sensor datato be used in accordance with an illustrative embodiment; and

FIG. 18 is a flowchart illustrating a process for sensor data fusion inaccordance with an illustrative embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Embodiments of this invention provide systems and methods for vehiclenavigation and more particularly systems and methods for a distributedknowledge base within a vehicle for controlling operation of a vehicle.As an example, embodiments of this invention provide a method and systemfor utilizing a versatile robotic control module for localization andnavigation of a vehicle.

Robotic or autonomous vehicles, sometimes referred to as mobile roboticplatforms, generally have a robotic control system that controls theoperational systems of the vehicle. In a vehicle that is limited to atransportation function, the operational systems may include steering,braking, transmission, and throttle systems. Such autonomous vehiclesgenerally have a centralized robotic control system for control of theoperational systems of the vehicle. Some military vehicles have beenadapted for autonomous operation. In the United States, some tanks,personnel carriers, Stryker vehicles, and other vehicles have beenadapted for autonomous capability. Generally, these are to be used in amanned mode as well.

The different illustrative embodiments recognize that robotic controlsystem sensor inputs may include data associated with the vehicle'sdestination, preprogrammed path information, and detected obstacleinformation. Based on such data associated with the information above,the vehicle's movements are controlled. Obstacle detection systemswithin a vehicle commonly use scanning lasers to scan a beam over afield of view, or cameras to capture images over a field of view. Thescanning laser may cycle through an entire range of beam orientations,or provide random access to any particular orientation of the scanningbeam. The camera or cameras may capture images over the broad field ofview, or of a particular spectrum within the field of view. For obstacledetection applications of a vehicle, the response time for collectingimage data should be rapid over a wide field of view to facilitate earlyrecognition and avoidance of obstacles.

Location sensing devices include odometers, global positioning systems,and vision-based triangulation systems. Many location sensing devicesare subject to errors in providing an accurate location estimate overtime and in different geographic positions. Odometers are subject tomaterial errors due to surface terrain. Satellite-based guidancesystems, such as global positioning system-based guidance systems, whichare commonly used today as a navigation aid in cars, airplanes, ships,computer-controlled harvesters, mine trucks, and other vehicles, mayexperience difficulty guiding when heavy foliage or other permanentobstructions, such as mountains, buildings, trees, and terrain, preventor inhibit global positioning system signals from being accuratelyreceived by the system. Vision-based triangulation systems mayexperience error over certain angular ranges and distance ranges becauseof the relative position of cameras and landmarks.

The illustrative embodiments also recognize that in order to provide asystem and method where a combination manned/autonomous vehicleaccurately navigates and manages a work-site, specific mechanicalaccommodations for processing means and location sensing devices arerequired. Therefore, it would be advantageous to have a method andapparatus to provide additional features for the navigation of vehicles.

With reference to the figures and in particular with reference to FIG.1, embodiments of the present invention may be used in a variety ofvehicles, such as automobiles, trucks, harvesters, combines,agricultural equipment, tractors, mowers, armored vehicles, and utilityvehicles. Embodiments of the present invention may also be used in asingle computing system or a distributed computing system. Theillustrative embodiments are not meant to limit the present invention inany way. FIG. 1 depicts a block diagram of multiple vehicles operatingin a network environment in accordance with an illustrative embodiment.FIG. 1 depicts an illustrative environment including network 100 in oneembodiment of the present invention. In this example, back office 102may be a single computer or a distributed computing cloud. Back office102 supports the physical databases and/or connections to externaldatabases which underlie the knowledge bases used in the differentillustrative embodiments. Back office 102 may supply knowledge bases todifferent vehicles, as well as provide online access to information fromknowledge bases. In this example, combine/harvesters 104, 106, and 108may be any type of harvesting, threshing, crop cleaning, or otheragricultural vehicle. In this illustrative embodiment,combine/harvesters 104, 106, and 108 operate on field 110, which may beany type of land used to cultivate crops for agricultural purposes.

In an illustrative example, combine/harvester 104 may move along field110 following a leader using a number of different modes of operation toaid an operator in performing agricultural tasks on field 110. The modesinclude, for example, a side following mode, a teach and playback mode,a teleoperation mode, a path mapping mode, a straight mode, and othersuitable modes of operation. An operator may be a person being followedas the leader when the vehicle is operating in a side-following mode, aperson driving the vehicle, or a person controlling the vehiclemovements in teleoperation mode. A leader may be a human operator oranother vehicle in the same worksite.

In one example, in the side following mode, combine/harvester 106 is theleader and combine/harvesters 104 and 108 are the followers. In anotherexample, in the side following mode an operator may be the leader andcombine/harvester 104 may be the follower. The side following mode mayinclude preprogrammed maneuvers in which an operator may change themovement of combine/harvester 104 from an otherwise straight travel pathfor combine/harvester 104. For example, if an obstacle is detected infield 110, the operator may initiate a go around obstacle maneuver thatcauses combine/harvester 104 to steer out and around an obstacle in apreset path. With this mode, automatic obstacle identification andavoidance features may still be used. With the teach and play back mode,for example, an operator may drive combine/harvester 104 along a path onfield 110 without stops, generating a mapped path. After driving thepath, the operator may move combine/harvester 104 back to the beginningof the mapped path. In the second pass on field 110, the operator maycause combine/harvester 104 to drive the mapped path from start point toend point without stopping, or may cause combine/harvester 104 to drivethe mapped path with stops along the mapped path. In this manner,combine/harvester 104 drives from start to finish along the mapped path.Combine/harvester 104 still may include some level of obstacle detectionto prevent combine/harvester 104 from running over or hitting anobstacle, such as a field worker or another agricultural vehicle, suchas combine/harvester 106 and 108.

In a teleoperation mode, for example, an operator may operate and/orwirelessly drive combine/harvester 104 across field 110 in a fashionsimilar to other remote controlled vehicles. With this type of mode ofoperation, the operator may control combine/harvester 104 through awireless controller.

In a path mapping mode, the different paths may be mapped by an operatorprior to reaching field 110. In a crop spraying example, routes may beidentical for each trip and the operator may rely on the fact thatcombine/harvester 104 will move along the same path each time.Intervention or deviation from the mapped path may occur only when anobstacle is present. Again, with the path mapping mode, way points maybe set to allow combine/harvester 104 to stop or turn at certain pointsalong field 110.

In a straight mode, combine/harvester 106 may be placed in the middle oroffset from some distance from a boundary, field edge, or other vehicleon field 110. In a grain harvesting example, combine/harvester 106 maymove down field 110 along a straight line allowing one or more othervehicles, such as combine/harvester 104 and 108, to travel in a parallelpath on either side of combine/harvester 106 to harvest rows of grain.In this type of mode of operation, the path of combine/harvester 106 isalways straight unless an obstacle is encountered. In this type of modeof operation, an operator may start and stop combine/harvester 106 asneeded. This type of mode may minimize the intervention needed by adriver.

In different illustrative embodiments, the different types of modes ofoperation may be used in combination to achieve the desired goals. Inthese examples, at least one of these modes of operation may be used tominimize driving while maximizing safety and efficiency in a harvestingprocess. In these examples, each of the different types of vehiclesdepicted may utilize each of the different types of modes of operationto achieve desired goals. As used herein the phrase “at least one of”when used with a list of items means that different combinations of oneor more of the items may be used and only one of each item in the listmay be needed. For example, “at least one of item A, item B, and item C”may include, for example, without limitation, item A or item A and itemB. This example also may include item A, item B, and item C or item Band item C. As another example, at least one of item A, item B, and itemC may include item A, two of item B, and 4 of item C or some othercombination types of items and/or number of items.

In different illustrative embodiments, dynamic conditions impact themovement of a vehicle. A dynamic condition is a change in theenvironment around a vehicle. For example, a dynamic condition mayinclude, without limitation movement of another vehicle in theenvironment to a new location, detection of an obstacle, detection of anew object or objects in the environment, receiving user input to changethe movement of the vehicle, receiving instructions from a back office,such as back office 102, and the like. In response to a dynamiccondition, the movement of a vehicle may be altered in various ways,including, without limitation stopping the vehicle, acceleratingpropulsion of the vehicle, decelerating propulsion of the vehicle, andaltering the direction of the vehicle, for example.

Further, autonomous routes may include several line segments. In otherexamples, a path may go around blocks in a square or rectangular patternor follow field contours or boundaries. Of course, other types ofpatterns also may be used depending upon the particular implementation.Routes and patterns may be performed with the aid of a knowledge base inaccordance with an illustrative embodiment. In these examples, anoperator may drive combine/harvester 104 onto a field or to a beginningposition of a path. The operator also may monitor combine/harvester 104for safe operation and ultimately provide overriding control for thebehavior of combine/harvester 104.

In these examples, a path may be a preset path, a path that iscontinuously planned with changes made by combine/harvester 104 tofollow a leader in a side following mode, a path that is directed by anoperator using a remote control in a teleoperation mode, or some otherpath. The path may be any length depending on the implementation. Pathsmay be stored and accessed with the aid of a knowledge base inaccordance with an illustrative embodiment.

In these examples, heterogeneous sets of redundant sensors are locatedon multiple vehicles in a worksite to provide high integrity perceptionwith fault tolerance. Redundant sensors in these examples are sensorsthat may be used to compensate for the loss and/or inability of othersensors to obtain information needed to control a vehicle. A redundantuse of the sensor sets are governed by the intended use of each of thesensors and their degradation in certain dynamic conditions. The sensorsets robustly provide data for localization and/or safeguarding in lightof a component failure or a temporary environmental condition. Forexample, dynamic conditions may be terrestrial and weather conditionsthat affect sensors and their ability to contribute to localization andsafeguarding. Such conditions may include, without limitation, sun,clouds, artificial illumination, full moon light, new moon darkness,degree of sun brightness based on sun position due to season, shadows,fog, smoke, sand, dust, rain, snow, and the like.

Thus, the different illustrative embodiments provide a number ofdifferent modes to operate a number of different vehicles, such ascombine/harvesters 104, 106, and 108. Although FIG. 1 illustrates avehicle for agricultural work, this illustration is not meant to limitthe manner in which different modes may be applied. For example, thedifferent illustrative embodiments may be applied to other types ofvehicles and other types of uses. As a specific example, the differentillustrative embodiments may be applied to a military vehicle in which asoldier uses a side following mode to provide a shield across aclearing. In other embodiments, the vehicle may be a compact utilityvehicle and have a chemical sprayer mounted and follow an operator asthe operator applies chemicals to crops or other foliage. These types ofmodes also may provide obstacle avoidance and remote controlcapabilities. As yet another example, the different illustrativeembodiments may be applied to delivery vehicles, such as those for thepost office or other commercial delivery vehicles. The illustrativeembodiments recognize a need for a system and method where a combinationmanned/autonomous vehicle can accurately navigate and manage awork-site. Therefore, the illustrative embodiments provide a computerimplemented method, apparatus, and computer program product forcontrolling a vehicle. A dynamic condition is identified using aplurality of sensors on the vehicle and the vehicle is controlled usinga knowledge base.

With reference now to FIGS. 2A and 2B, a block diagram illustratingvehicle perception used to adjust navigation is depicted in accordancewith an illustrative embodiment. Vehicles 200 and 206 are examples ofone or more of combine/harvesters 104, 106, and 108 in FIG. 1. Vehicle200 travels across terrain 202 using sensors located on vehicle 200 toperceive attributes of the terrain. In normal operating conditions, maxdetection range 204 of the sensors on vehicle 200 offers good visibilityof upcoming terrain in the path of vehicle 200. Vehicle 206 travelsacross terrain 208, which limits max detection range 210 and providesdiminished detection range 212. Terrain 208 may be, for example, astructure or vegetation obscuring visibility, land topography limitingthe sensors range of detection, and the like. Vehicle 206 may adjust thespeed and following distance based upon the detection range available.For example, when approaching terrain 208 with diminished detectionrange 212, vehicle 206 may slow its speed in order to increasesafeguarding capabilities, such as obstacle detection, obstacleavoidance, and emergency stopping. In an illustrative embodiment,vehicle 200 and vehicle 206 may be working in different areas of thesame worksite. When vehicle 206 experiences diminished detection range212, vehicle 206 may request sensor data information from vehicle 200.Sensor data is any data generated by a sensor. For example, diminisheddetection range 212 may be due to degradation of global positioningsystem capabilities based on the tree canopy of terrain 208. Vehicle200, however, may be operating on a parallel or nearby path within thesame worksite, but away from the tree canopy of terrain 208, withterrain 202 providing the global positioning system receiver located onvehicle 200 an ability to receive signals. The sensor system of vehicle200 may determine a position estimate for vehicle 200, and a relativeposition estimate of vehicle 206 based on other sensors detecting thedistance, speed, and location of vehicle 206. Vehicle 200 may thentransmit localization information to vehicle 206, and vehicle 206 mayuse the information from the sensor system of vehicle 200 to determine aposition estimate for vehicle 206 and thereby maintain vehicle speed andprogression along the planned path.

With reference now to FIG. 3, a block diagram of components used tocontrol a vehicle is depicted in accordance with an illustrativeembodiment. In this example, vehicle 300 is an example of a vehicle,such as combine/harvesters 104, 106, and 108 in FIG. 1. Vehicle 300 isalso an example of vehicle 200 and vehicle 206 in FIGS. 2A and 2B. Inthis example, vehicle 300 includes machine controller 302, steeringsystem 304, braking system 306, propulsion system 308, sensor system310, and communication unit 312.

Machine controller 302 may be, for example, a data processing system orsome other device that may execute processes to control movement of avehicle. Machine controller 302 may be, for example, a computer, anapplication integrated specific circuit, or some other suitable device.Machine controller 302 may execute processes to control steering system304, braking system 306, and propulsion system 308 to control movementof the vehicle. Machine controller 302 may send various commands tothese components to operate the vehicle in different modes of operation.These commands may take various forms depending on the implementation.For example, the commands may be analog electrical signals in which avoltage and/or current change is used to control these systems. In otherimplementations, the commands may take the form of data sent to thesystems to initiate the desired actions. Steering system 304 may controlthe direction or steering of the vehicle in response to commandsreceived from machine controller 302. Steering system 304 may be, forexample, an electrically controlled hydraulic steering system, anelectrically driven rack and pinion steering system, an Ackermansteering system, or some other suitable steering system. Braking system306 may slow down and/or stop the vehicle in response to commands frommachine controller 302. Braking system 306 may be an electricallycontrolled braking system. This braking system may be, for example, ahydraulic braking system, a friction braking system, or some othersuitable braking system that may be electrically controlled.

In these examples, propulsion system 308 may propel or move the vehiclein response to commands from machine controller 302. Propulsion system308 may maintain or increase the speed at which a vehicle moves inresponse to instructions received from machine controller 302.Propulsion system 308 may be an electrically controlled propulsionsystem. Propulsion system 308 may be, for example, an internalcombustion engine, an internal combustion engine/electric hybrid system,an electric engine, or some other suitable propulsion system. Sensorsystem 310 may be a set of sensors used to collect information about theenvironment around vehicle 300. This information collected by sensorsystem 310 may be used for localization in identifying a location ofvehicle 300 or a location of another vehicle in the environment. Inthese examples, the information is sent to machine controller 302 toprovide data in identifying how the vehicle should move in differentmodes of operation. For example, braking system 306 may slow vehicle 300in response to a limited detection range of sensor system 310 on vehicle300, such as diminished detection range 212 in FIG. 2B. In theseexamples, a set refers to one or more items. A set of sensors is one ormore sensors in these examples. Communication unit 312 may providecommunications links to machine controller 302 to receive information.This information includes, for example, data, commands, and/orinstructions. Communication unit 312 may take various forms. Forexample, communication unit 312 may include a wireless communicationssystem, such as a cellular phone system, a Wi-Fi wireless system, aBluetooth wireless system, or some other suitable wirelesscommunications system. Further, communication unit 312 also may includea communications port, such as, for example, a universal serial busport, a serial interface, a parallel port interface, a networkinterface, or some other suitable port to provide a physicalcommunications link. Communication unit 312 may be used to communicatewith a remote location or an operator. Communications unit 312 mayinclude a battery back-up on a plurality of electronic modules that eachoperates at a different frequency in order to minimize the likelihood ofcommon mode failure.

With reference now to FIG. 4, a block diagram of a data processingsystem is depicted in accordance with an illustrative embodiment. Dataprocessing system 400 is an example of one manner in which machinecontroller 302 in FIG. 3 may be implemented. In this illustrativeexample, data processing system 400 includes communications fabric 402,which provides communication between processor unit 404, memory 406,persistent storage 408, communications unit 410, input/output (I/O) unit412, and display 414. Processor unit 404 serves to execute instructionsfor software that may be loaded into memory 406. Processor unit 404 maybe a set of one or more processors or may be a multi-processor core,depending on the particular implementation. Further, processor unit 404may be implemented using one or more heterogeneous processor systems inwhich a main processor is present with secondary processors on a singlechip. As another illustrative example, processor unit 404 may be asymmetric multi-processor system containing multiple processors of thesame type. Memory 406 and persistent storage 408 are examples of storagedevices. A storage device is any piece of hardware that is capable ofstoring information either on a temporary basis and/or a permanentbasis. Memory 406, in these examples, may be, for example, a randomaccess memory or any other suitable volatile or non-volatile storagedevice. Persistent storage 408 may take various forms depending on theparticular implementation. For example, persistent storage 408 maycontain one or more components or devices. For example, persistentstorage 408 may be a hard drive, a flash memory, a rewritable opticaldisk, a rewritable magnetic tape, or some combination of the above. Themedia used by persistent storage 408 also may be removable. For example,a removable hard drive may be used for persistent storage 408.Communications unit 410, in these examples, provides for communicationswith other data processing systems or devices. In these examples,communications unit 410 is a network interface card. Communications unit410 may provide communications through the use of either or bothphysical and wireless communications links.

Input/output unit 412 allows for input and output of data with otherdevices that may be connected to data processing system 400. Forexample, input/output unit 412 may provide a connection for user inputthrough a keyboard and mouse. Further, input/output unit 412 may sendoutput to a printer. Display 414 provides a mechanism to displayinformation to a user. Instructions for the operating system andapplications or programs are located on persistent storage 408. Theseinstructions may be loaded into memory 406 for execution by processorunit 404. The processes of the different embodiments may be performed byprocessor unit 404 using computer implemented instructions, which may belocated in a memory, such as memory 406. These instructions are referredto as program code, computer usable program code, or computer readableprogram code that may be read and executed by a processor in processorunit 404. The program code in the different embodiments may be embodiedon different physical or tangible computer readable media, such asmemory 406 or persistent storage 408. Program code 416 is located in afunctional form on computer readable media 418 that is selectivelyremovable and may be loaded onto or transferred to data processingsystem 400 for execution by processor unit 404. Program code 416 andcomputer readable media 418 form computer program product 420 in theseexamples. In one example, computer readable media 418 may be in atangible form, such as, for example, an optical or magnetic disc that isinserted or placed into a drive or other device that is part ofpersistent storage 408 for transfer onto a storage device, such as ahard drive that is part of persistent storage 408. In a tangible form,computer readable media 418 also may take the form of a persistentstorage, such as a hard drive, a thumb drive, or a flash memory that isconnected to data processing system 400. The tangible form of computerreadable media 418 is also referred to as computer recordable storagemedia. In some instances, computer readable media 418 may not beremovable. Alternatively, program code 416 may be transferred to dataprocessing system 300 from computer readable media 418 through acommunications link to communications unit 410 and/or through aconnection to input/output unit 412. The communications link and/or theconnection may be physical or wireless in the illustrative examples. Thecomputer readable media also may take the form of non-tangible media,such as communications links or wireless transmissions containing theprogram code.

The different components illustrated for data processing system 400 arenot meant to provide architectural limitations to the manner in whichdifferent embodiments may be implemented. The different illustrativeembodiments may be implemented in a data processing system includingcomponents in addition to or in place of those illustrated for dataprocessing system 400. Other components shown in FIG. 4 can be variedfrom the illustrative examples shown. As one example, a storage devicein data processing system 400 is any hardware apparatus that may storedata. Memory 406, persistent storage 408, and computer readable media418 are examples of storage devices in a tangible form.

In another example, a bus system may be used to implement communicationsfabric 402 and may be comprised of one or more buses, such as a systembus or an input/output bus. Of course, the bus system may be implementedusing any suitable type of architecture that provides for a transfer ofdata between different components or devices attached to the bus system.Additionally, a communications unit may include one or more devices usedto transmit and receive data, such as a modem or a network adapter.Further, a memory may be, for example, memory 406 or a cache, such asfound in an interface and memory controller hub that may be present incommunications fabric 402.

With reference now to FIG. 5, a block diagram of a sensor system isdepicted in accordance with an illustrative embodiment. Sensor system500 is an example of one implementation of sensor system 310 in FIG. 3.Sensor system 500 includes redundant sensors. A redundant sensor inthese examples is a sensor that may be used to compensate for the lossand/or inability of another sensor to obtain information needed tocontrol a vehicle. A redundant sensor may be another sensor of the sametype (homogenous) and/or a different type of sensor (heterogeneous) thatis capable of providing information for the same purpose as the othersensor.

As illustrated, sensor system 500 includes, for example, globalpositioning system 502, structured light sensor 504, twodimensional/three dimensional lidar 506, dead reckoning 508, infraredcamera 510, visible light camera 512, radar 514, ultrasonic sonar 516,radio frequency identification reader 518, rain sensor 520, and ambientlight sensor 522. These different sensors may be used to identify theenvironment around a vehicle. For example, these sensors may be used todetect terrain in the path of a vehicle, such as terrain 202 and 208 inFIGS. 2A and 2B. In another example, these sensors may be used to detecta dynamic condition in the environment. The sensors in sensor system 500may be selected such that one of the sensors is always capable ofsensing information needed to operate the vehicle in different operatingenvironments. Global positioning system 502 may identify the location ofthe vehicle with respect to other objects in the environment. Globalpositioning system 502 may be any type of radio frequency triangulationscheme based on signal strength and/or time of flight. Examples include,without limitation, the Global Positioning System, Glonass, Galileo, andcell phone tower relative signal strength. Position is typicallyreported as latitude and longitude with an error that depends onfactors, such as ionispheric conditions, satellite constellation, andsignal attenuation from vegetation.

Structured light sensor 504 emits light in a pattern, such as one ormore lines, reads back the reflections of light through a camera, andinterprets the reflections to detect and measure objects in theenvironment. Two dimensional/three dimensional lidar 506 is an opticalremote sensing technology that measures properties of scattered light tofind range and/or other information of a distant target. Twodimensional/three dimensional lidar 506 emits laser pulses as a beam,than scans the beam to generate two dimensional or three dimensionalrange matrices. The range matrices are used to determine distance to anobject or surface by measuring the time delay between transmission of apulse and detection of the reflected signal.

Dead reckoning 508 begins with a known position, which is then advanced,mathematically or directly, based upon known speed, elapsed time, andcourse. The advancement based upon speed may use the vehicle odometer,or ground speed radar, to determine distance traveled from the knownposition. Infrared camera 510 detects heat indicative of a living thingversus an inanimate object. An infrared camera may also form an imageusing infrared radiation. Visible light camera 512 may be a standardstill-image camera, which may be used alone for color information orwith a second camera to generate stereoscopic or three-dimensionalimages. When visible light camera 512 is used along with a second camerato generate stereoscopic images, the two or more cameras may be set withdifferent exposure settings to provide improved performance over a rangeof lighting conditions. Visible light camera 512 may also be a videocamera that captures and records moving images. Radar 514 useselectromagnetic waves to identify the range, altitude, direction, orspeed of both moving and fixed objects. Radar 514 is well known in theart, and may be used in a time of flight mode to calculate distance toan object, as well as Doppler mode to calculate the speed of an object.Ultrasonic sonar 516 uses sound propagation on an ultrasonic frequencyto measure the distance to an object by measuring the time fromtransmission of a pulse to reception and converting the measurement intoa range using the known speed of sound. Ultrasonic sonar 516 is wellknown in the art and can also be used in a time of flight mode orDoppler mode, similar to radar 514. Radio frequency identificationreader 518 relies on stored data and remotely retrieves the data usingdevices called radio frequency identification (RFID) tags ortransponders. Rain sensor 520 detects precipitation on an exteriorsurface of the vehicle. Ambient light sensor 522 measures the amount ofambient light in the environment.

Sensor system 500 may retrieve environmental data from one or more ofthe sensors to obtain different perspectives of the environment. Forexample, sensor system 500 may obtain visual data from visible lightcamera 512, data about the distance of the vehicle in relation toobjects in the environment from two dimensional/three dimensional lidar506, and location data of the vehicle in relation to a map from globalpositioning system 502.

Sensor system 500 is capable of detecting objects even in differentoperating environments. For example, global positioning system 502 maybe used to identify a position of the vehicle. If a field is surroundedby trees with thick canopies during the spring, global positioningsystem 502 may be unable to provide location information on some areasof the field. In this situation, visible light camera 512 and/ortwo-dimensional/three-dimensional lidar 506 may be used to identify alocation of the vehicle relative to non-mobile objects, such astelephone poles, trees, roads and other suitable landmarks.

In addition to receiving different perspectives of the environment,sensor system 500 provides redundancy in the event of a sensor failure,which facilitates high-integrity operation of the vehicle. For example,in an illustrative embodiment, if visible light camera 512 is theprimary sensor used to identify the location of the operator inside-following mode, and visible light camera 512 fails, radio frequencyidentification reader 518 will still detect the location of the operatorthrough a radio frequency identification tag worn by the operator,thereby providing redundancy for safe operation of the vehicle.

With reference now to FIG. 6, a block diagram of functional softwarecomponents that may be implemented in a machine controller is depictedin accordance with an illustrative embodiment. In this example,different functional software components that may be used to control avehicle are illustrated. The vehicle may be a vehicle, such ascombine/harvester 104, 106, and 108 in FIG. 1. Machine controller 600may be implemented in a vehicle, such as vehicle 200 and vehicle 206 inFIGS. 2A and 2B or vehicle 300 in FIG. 3 using a data processing system,such as data processing system 400 in FIG. 4. In this example machinecontrol process 602, sensor processing algorithms 604, user interface606, knowledge base 608, behavior library 610, knowledge base process612, and object anomaly rules 616 are present in machine controller 600.

Machine control process 602 transmits signals to steering, braking, andpropulsion systems, such as steering system 304, braking system 306, andpropulsion system 308 in FIG. 3. Machine control process 602 may alsotransmit signals to components of a sensor system, such as sensor system500 in FIG. 5. For example, in an illustrative embodiment, machinecontrol process 602 transmits a signal to a camera component of sensorsystem 500 in order to pan, tilt, or zoom a lens of the camera toacquire different images and perspectives of an environment around thevehicle. Machine control process 602 may also transmit signals tosensors within sensor system 500 in order to activate, deactivate, ormanipulate the sensor itself. Sensor processing algorithms 604 receivessensor data from sensor system 500 and classifies the sensor data intothematic features. This classification may include identifying objectsthat have been detected in the environment. For example, sensorprocessing algorithms 604 may classify an object as a person, telephonepole, tree, road, light pole, driveway, fence, or some other type ofobject. The classification may be performed to provide information aboutobjects in the environment. This information may be used to generate athematic map, which may contain a spatial pattern of attributes. Theattributes may include classified objects. The classified objects mayinclude dimensional information, such as, for example, location, height,width, color, and other suitable information. This map may be used toplan actions for the vehicle. The action may be, for example, planningpaths to follow an operator in a side following mode or performingobject avoidance.

The classification may be done autonomously or with the aid of userinput through user interface 606. For example, in an illustrativeembodiment, sensor processing algorithms 604 receives data from a laserrange finder, such as two dimensional/three dimensional lidar 506 inFIG. 5, identifying points in the environment. User input may bereceived to associate a data classifier with the points in theenvironment, such as, for example, a data classifier of “tree”associated with one point, and “fence” with another point. Tree andfence are examples of thematic features in an environment. Sensorprocessing algorithms 604 then interacts with knowledge base 608 tolocate the classified thematic features on a thematic map stored inknowledge base 608, and calculates the vehicle position based on thesensor data in conjunction with the landmark localization. Machinecontrol process 602 receives the environmental data from sensorprocessing algorithms 604, and interacts with knowledge base 608 andbehavior library 610 in order to determine which commands to send to thevehicle's steering, braking, and propulsion components.

Sensor processing algorithms 604 analyzes sensor data for accuracy andfuses selected sensor data to provide a single value that may be sharedwith other machines. Analyzing the sensor data for accuracy involvesdetermining an accuracy level for the sensor data based on the sensordata relative to other sensor data and the confidence level in thesensor. For example, with global positioning data from a globalpositioning system receiver, the reported change in position in latitudeand longitude may be compared with radar and wheel-based odometry. Ifthe global positioning system distance is a certain percentage differentfrom two close values from other sources, it is considered an outlier.The distance may also be compared to a theoretical maximum distance avehicle could move in a given unit of time. Alternately, the currentsatellite geometric dilution of precision could be used to validate thelatitude and longitude for use in further computations. The accuracylevel will influence which sensor data is fused and which sensor data isconsidered an outlier. Outliers are determined using statistical methodscommonly known in the field of statistics. Sensor data is fused bymathematically processing the sensor data to obtain a single value usedto determine relative position. Examples of this mathematical processinginclude, but are not limited to, simple averaging, weighted averaging,and median filtering. Component failures of a sensor system on a vehiclecan then be detected by comparing the position and environmentinformation provided by each sensor or fused set of sensors. Forexample, if a sensor is out of a margin of error for distance, angle,position, and the like, it is likely that the sensor has failed or iscompromised and should be removed from the current calculation. Repeatedexcessive errors are grounds for declaring the sensor failed until acommon mode root cause is eliminated, or until the sensor is repaired orreplaced. In an illustrative embodiment, a global positioning system,such as global positioning system 502 of sensor system 500 in FIG. 5, ona vehicle, such as combine/harvester 106 in FIG. 1, determines its ownposition. Furthermore, it detects the position of another vehicle, suchas combine/harvester 104, as being fifty feet ahead and thirty degreesto its left. One visible light camera, such as visible light camera 512in FIG. 5, on combine/harvester 106 detects combine/harvester 104 asbeing forty-eight feet ahead and twenty-eight degrees left, whileanother visible light camera on combine/harvester 106 detectscombine/harvester 104 as being forty-nine feet ahead and twenty-ninedegrees left. A lidar, such as two dimensional/three dimensional lidar506 in FIG. 5, on combine/harvester 106 detects combine/harvester 104 asbeing fifty-one feet ahead and thirty-one degrees left. Sensorprocessing algorithms 604 receives the sensor data from the globalpositioning system, visible light cameras, and lidar, and fuses themtogether using a simple average of distances and angles to determine therelative position of combine/harvester 104 as being 49.5 feet ahead and29.5 degrees left.

These illustrative examples are not meant to limit the invention in anyway. Multiple types of sensors and sensor data may be used to performmultiple types of localization. For example, the sensor data may befused to determine the location of an object in the environment, or forobstacle detection. Sensor data analysis and fusion may also beperformed by machine control process 602 in machine controller 600.

Knowledge base 608 contains information about the operating environment,such as, for example, a fixed map showing streets, structures, treelocations, and other static object locations. Knowledge base 608 mayalso contain information, such as, without limitation, local flora andfauna of the operating environment, current weather for the operatingenvironment, weather history for the operating environment, specificenvironmental features of the work area that affect the vehicle, and thelike. The information in knowledge base 608 may be used to performclassification and plan actions. Knowledge base 608 may be locatedentirely in machine controller 600 or parts or all of knowledge base 608may be located in a remote location that is accessed by machinecontroller 600. Behavior library 610 contains various behavioralprocesses specific to machine coordination that can be called andexecuted by machine control process 602. In one illustrative embodiment,there may be multiple copies of behavior library 610 on machinecontroller 600 in order to provide redundancy. The library is accessedby machine control process 602.

Knowledge base process 612 interacts with sensor processing algorithms604 to receive processed sensor data about the environment, and in turninteracts with knowledge base 608 to classify objects detected in theprocessed sensor data. Knowledge base process 612 also informs machinecontrol process 602 of the classified objects in the environment inorder to facilitate accurate instructions for machine control process602 to send to steering, braking, and propulsion systems. For example,in an illustrative embodiment, sensor processing algorithms 604 detectstall, narrow, cylindrical objects along the side of the planned path.Knowledge base process 612 receives the processed data from sensorprocessing algorithms 604 and interacts with knowledge base 608 toclassify the tall, narrow, cylindrical objects as tree trunks. Knowledgebase process 612 can then inform machine control process 602 of thelocation of the tree trunks in relation to the vehicle, as well as anyfurther rules that may apply to tree trunks in association with theplanned path.

Object anomaly rules 616 provide machine control process 602instructions on how to operate the vehicle when an anomaly occurs, suchas sensor data received by sensor processing algorithms 604 beingincongruous with environmental data stored in knowledge base 608. Forexample, object anomaly rules 616 may include, without limitation,instructions to alert the operator via user interface 606 orinstructions to activate a different sensor in sensor system 500 inorder to obtain a different perspective of the environment.

With reference now to FIG. 7, a block diagram of a knowledge base isdepicted in accordance with an illustrative embodiment. Knowledge base700 is an example of a knowledge base component of a machine controller,such as knowledge base 608 of machine controller 600 in FIG. 6. Forexample, knowledge base 700 may be, without limitation, a component of anavigation system, an autonomous machine controller, a semi-autonomousmachine controller, or may be used to make management decisionsregarding work-site activities. Knowledge base 700 includes a prioriknowledge base 702, online knowledge base 704, and learned knowledgebase 706.

A priori knowledge base 702 contains static information about theoperating environment of a vehicle. Types of information about theoperating environment of a vehicle may include, without limitation, afixed map showing streets, structures, trees, and other static objectsin the environment; stored geographic information about the operatingenvironment; and weather patterns for specific times of the yearassociated with the operating environment. A priori knowledge base 702may also contain fixed information about objects that may be identifiedin an operating environment, which may be used to classify identifiedobjects in the environment. This fixed information may includeattributes of classified objects, for example, an identified object withattributes of tall, narrow, vertical, and cylindrical, may be associatedwith the classification of “telephone pole.” A priori knowledge base 702may further contain fixed work-site information. A priori knowledge base702 may be updated based on information from online knowledge base 704,and learned knowledge base 706.

Online knowledge base 704 may be accessed with a communications unit,such as communications unit 312 in FIG. 3, to wirelessly access theInternet. Online knowledge base 704 dynamically provides information toa machine control process which enables adjustment to sensor dataprocessing, site-specific sensor accuracy calculations, and/or exclusionof sensor information. For example, online knowledge base 704 mayinclude current weather conditions of the operating environment from anonline source. In some examples, online knowledge base 704 may be aremotely accessed knowledge base. This weather information may be usedby machine control process 602 in FIG. 6 to determine which sensors toactivate in order to acquire accurate environmental data for theoperating environment. Weather, such as rain, snow, fog, and frost maylimit the range of certain sensors, and require an adjustment inattributes of other sensors in order to acquire accurate environmentaldata from the operating environment. Other types of information that maybe obtained include, without limitation, vegetation information, such asfoliage deployment, leaf drop status, and lawn moisture stress, andconstruction activity, which may result in landmarks in certain regionsbeing ignored. In another illustrative environment, online knowledgebase 704 may be used to note when certain activities are in process thataffect operation of sensor processing algorithms in machine controller600. For example, if tree pruning is in progress, a branch matchingalgorithm should not be used, but a tree trunk matching algorithm maystill be used, as long as the trees are not being cut down completely.When the machine controller receives user input signaling that thepruning process is over, the sensor system may collect environmentaldata to analyze and update a priori knowledge base 702.

Learned knowledge base 706 may be a separate component of knowledge base700, or alternatively may be integrated with a priori knowledge base 702in an illustrative embodiment. Learned knowledge base 706 containsknowledge learned as the vehicle spends more time in a specific workarea, and may change temporarily or long-term depending uponinteractions with online knowledge base 704 and user input. For example,learned knowledge base 706 may detect the absence of a tree that waspresent the last time it received environmental data from the work area.Learned knowledge base 706 may temporarily change the environmental dataassociated with the work area to reflect the new absence of a tree,which may later be permanently changed upon user input confirming thetree was in fact cut down. Learned knowledge base 706 may learn throughsupervised or unsupervised learning.

With reference now to FIG. 8, a block diagram of a format in a knowledgebase used to select sensors for use in planning paths and obstacleavoidance is depicted in accordance with an illustrative embodiment.This format may be used by knowledge base process 612 and machinecontrol process 602 in FIG. 6. The format is depicted in sensor table800 illustrating heterogeneous sensor redundancy for localization of avehicle on a street. This illustrative embodiment is not meant to limitthe present invention in any way. Other illustrative embodiments may usethis format for localization of a vehicle on a field, golf course,off-road terrain, and other geographical areas.

Global positioning systems 802 would likely not have real time kinematicaccuracy in a typical street environment due to structures andvegetation. Normal operating conditions 804 would provide good to poorquality signal reception 806 because the global positioning systemsignal reception quality would depend upon the thickness of the treecanopy over the street. In early fall 808, when some leaves are still onthe trees and others are filling the gutter or ditch alongside the road,the canopy thickness may offer good to poor quality signal reception810. However, in winter 812, when trees other than evergreens tend tohave little to no leaves, signal reception may be good to very good 814.Visible camera images of a curb or street edge 816 might offer excellentquality images 818 in normal operating conditions 804. However, in earlyfall 808 and winter 812, when leaves or snow obscure curb or street edgevisibility, visible camera images would offer unusable quality images820 and 822. Visible camera images 824 of the area around the vehicle,with an image height of eight feet above the ground, would offerexcellent quality images 826, 828, and 830 in most seasons, althoughweather conditions, such as rain or fog may render the images unusable.Landmarks identified at eight feet above the ground include objects,such as, without limitation, houses, light poles, and tree trunks. Thisheight is typically below tree canopies and above transient objects,such as cars, people, bikes, and the like, and provides a quality zonefor static landmarks. Visible camera images of the street crown 832 mayoffer good quality images 834 in normal operating conditions 804. Thestreet crown is typically the center of the street pavement, and imagesof the pavement may be used in a pavement pattern matching program forvehicle localization. In early fall 808, when leaves begin to fall andpartially obscure the pavement, visible camera images of the streetcrown 832 may be good to poor quality images 836 depending on the amountof leaves on the ground. In winter 812, the visible camera images of thestreet crown 832 may be unusable quality images 838 due to fresh snowobscuring the pavement. Lidar images of a curb 840 using pulses of lightmay be excellent 842 for detecting a curb or ground obstacle in normaloperating conditions 804, but may be unusable 844 when curb visibilityis obscured by leaves in early fall 808 or snow in winter 812. Lidardetection of the area eight feet above the ground 846 around the vehiclemay be excellent 848 in normal operating conditions 804, early fall 808,and winter 812, because the landmarks, such as houses and tree trunks,are not obscured by falling leaves or fresh snow. Lidar images of thesky 850 captures limb patterns above the street for use in limb patternmatching for vehicle localization. Lidar images of the sky 850 would beunusable due to the canopy 852 in normal operating conditions 804, andunusable to poor 854 in the early fall 808 when the majority of leavesremain on the limbs. However, lidar images of the sky 850 may beexcellent 856 in winter 812 when limbs are bare.

In another illustrative example, a group of three coordinated combinesmay be tasked with harvesting a crop, such as combine/harvester 104,106, and 108 on field 110 in FIG. 1. In this example, multiple vehiclesare working together, potentially operating in close proximity on field110. The worksite, field 110, may have few fixed visual landmarks, suchas telephone poles, for task-relative localization. In this example,communication between combine/harvester 104, 106, and 108 is importantfor vehicle-relative localization. The goals for combine/harvester 104,106, and 108 may be the following: not to harm people, property, orself; not to skip any of the crop for harvesting; perform efficientlywith minimum overlap between passes; and perform efficiently withoptimal coordination between vehicles. In order to meet these goals inview of the work-site environment, a combination of sensors, such asglobal positioning system 502, visible light camera 512, and twodimensional/three dimensional lidar 506 in FIG. 5 may be used toestimate vehicle position relative to other vehicles on the worksite andmaintain a safe distance between each vehicle.

With reference now to FIG. 9, a flowchart illustrating a process forsensor selection is depicted in accordance with an illustrativeembodiment. This process may be executed by knowledge base process 612in FIG. 6 or by sensor processing algorithms 604 in FIG. 6.

The process begins by retrieving the sensor table (step 902), such assensor table 800 in FIG. 8. The process identifies operating conditions(step 904) in the operating environment through sensor data receivedfrom a sensor system, such as sensor system 500 in FIG. 5 or onlineknowledge base 704 in FIG. 7. The process then determines whether theoperating conditions in the sensor table correspond with sensor datafrom the operating environment (step 906). If the sensor data from theenvironment does not correspond to the preset operating conditions inthe sensor table, the process adjusts the operating conditionsaccordingly (step 908) and selects the sensors to activate (step 910),with the process terminating thereafter. If the process determines thatthe sensor data corresponds with the sensor table information, theprocess moves directly to select the sensors to activate (step 910),with the process terminating thereafter.

With reference now to FIG. 10, a flowchart illustrating a process forprioritizing sensor data is depicted in accordance with an illustrativeembodiment. This process may be executed by machine controller 302 inFIG. 3. The process begins by receiving information about theenvironment from sensor data (step 1002) and comparing the informationabout the environment received from sensor data with the sensor table(step 1004) stored in a priori knowledge base 702 in FIG. 7. Next, theprocess calculates the differences and similarities in the informationand the sensor table (step 1006) and assigns an a priori weighting tothe data based on the calculation (step 1008), with the processterminating thereafter.

For example, the vehicle may be operating in normal operating conditionsmode, with the online knowledge base indicating it is spring. The sensortable may indicate that the appropriate sensors for normal operatingconditions are a visible light camera with a view of the street crown,and a visible light camera with a view of the road edge. However,information about the environment received from the sensors may indicatethat snow is covering the ground and obscuring the street and curb orroad edge, perhaps due to a late snow. The detection of snow may beverified by accessing the online knowledge base for current weatherconditions. As a result, the sensor data indicating the presence of snowmay be weighed more heavily than information from the sensor table aboutnormal operating conditions, with the sensors chosen to activateadjusted accordingly.

With reference now to FIG. 11, a flowchart illustrating a process forobject classification is depicted in accordance with an illustrativeembodiment. This process may be implemented by knowledge base process612 in FIG. 6. The process begins by receiving sensor data detecting anobject in the operating environment (step 1102). For example, withoutlimitation, an object might be a tree, light pole, person, animal,vehicle, and the like. Next, the process retrieves object identificationinformation from an object database (step 1104) located in a prioriknowledge base 702 in FIG. 7. Object information includes attributes andclassifiers for objects detected in an operating environment. Attributesmay include, for example, features such as tall, narrow, vertical,cylindrical, smooth texture, rough texture, bark texture, branches,leaves, no branches, no leaves, short, color, four to six inch highvertical, interruption in four to six inch high vertical, and the like.Classifiers may include, for example, tree, light pole, fire hydrant,curb, driveway, street, waste container, house, garage door, and thelike. The process classifies the object detected using objectidentification information and sensor data (step 1106), with the processterminating thereafter.

With reference now to FIG. 12, a flowchart illustrating a process forprocessing object anomalies is depicted in accordance with anillustrative embodiment. The process illustrated in FIG. 12 may beimplemented in a software component, such as machine control process602, using object anomaly rules 616 in FIG. 6. The process begins byselecting sensor data regarding the operating environment (step 1202).The process performs localization (step 1204) based on the sensor data,and generates a map of the operating environment (step 1206). The mapmay be generated by accessing a fixed map and route database of an apriori knowledge base, such as a priori knowledge base 702 in FIG. 7,and retrieving the map associated with the location of the vehicle asidentified by the sensor system, such as sensor system 500 in FIG. 5.Next, the process determines whether object anomalies are present (step1208). An object anomaly may be, for example, the presence of an objectthat is unaccounted for, the presence of an object that is not yetidentified, a change in a previously identified object, the absence ofan object, and the like. If anomalies are not present, the processreturns to select sensor data regarding the operating environment (step1202). If anomalies are present, the process processes the anomalies(step 1210), and returns to select sensor data regarding the operatingenvironment (step 1202). Processing anomalies may include updating oneor more knowledge bases, such as learned knowledge base 706 in FIG. 7,with the object anomaly information.

With reference now to FIG. 13, a flowchart illustrating a process forgenerating a thematic map is depicted in accordance with an illustrativeembodiment. The process illustrated in FIG. 13 may be implemented in asoftware component, such as knowledge base process 612 in FIG. 6. Theprocess begins by receiving sensor data regarding the operatingenvironment (step 1302). The process identifies superior sensorprocessing algorithms for the operating conditions (step 1304) andperforms localization (step 1306). Next, the process retrieves a staticmap of the environment (step 1308) from a fixed map/route database in apriori knowledge base 702 in FIG. 7 for example. The process receivessensor data detecting objects in the operating environment (step 1310)from a sensor system, such as sensor system 500 in FIG. 5. The processthen classifies the objects and populates the static map with thedetected classified objects (step 1312) in order to form a thematic map,with the process terminating thereafter. The thematic map may be storedin a knowledge base, such as knowledge base 608 in FIG. 6 and used bymachine control process 602 in FIG. 6 to execute a planned path whileavoiding obstacles identified in the thematic map.

With reference now to FIG. 14, a flowchart illustrating a process formonitoring sensor integrity is depicted in accordance with anillustrative embodiment. This process may be implemented by machinecontroller 302 in FIG. 3. The process begins by selecting sensors thatcorrespond with the planned path (step 1402). For example, a plannedpath of a residential street may correspond with a visible camera sensorduring summer months when the curb is clearly visible. Next, the processactivates the selected sensors (step 1404) and monitors for sensorfailure (step 1406). When the process detects incongruous informationfrom a sensor (step 1408), the process determines whether the sensor isin error or failure (step 1410). If the sensor is in error or failure,the process selects an alternate sensor (step 1412), and continues tomonitor for sensor failure (step 1406). If the sensor is not in error orfailure, the process generates an alert (step 1414), with the processterminating thereafter.

With reference now to FIG. 15, a flowchart illustrating a process forrequesting localization information from another vehicle is depicted inaccordance with an illustrative embodiment. The process may beimplemented by machine controller 302 utilizing communications unit 312in FIG. 3. This process may be implemented by a vehicle that is unableto obtain needed information, such as sensor data. In these examples,needed sensor data is any sensor data that is needed to control thevehicle. The sensor data may be, for example, data needed to performlocalization.

The process begins by transmitting a request for vehicle localizationinformation (step 1502) to other vehicles working in the same worksite.For example, in an illustrative embodiment, combine/harvester 106 inFIG. 1 may lose global positioning system capabilities and be unable todetermine a global position estimate. Combine/harvester 106 may transmitthe request for localization information to combine/harvester 104 and108 in order to utilize the sensor information detected from the sensorsystem on each of combine/harvester 104 and 108 respectively todetermine the position estimate of combine/harvester 106.

Next, the process receives information from other vehicles in theoperating environment (step 1504). The information may be referred to asalternate information and may include alternate sensor data. In otherexamples, the information also may include information from an onlineknowledge base that may not be reachable by the vehicle if acommunications unit has failed.

In an illustrative embodiment, the sensor information received fromother vehicles, such as combine/harvester 104 and 108 in FIG. 1, mayindicate the position estimate of each vehicle based on globalpositioning system information, as well as a relative position estimateof each vehicle in relation to the requesting vehicle, such ascombine/harvester 106 in FIG. 1. Combine/harvester 106 may then use theposition estimate of each vehicle and the relative position estimate ofeach vehicle in relation to combine/harvester 106 to determine aposition estimate of combine/harvester 106. Information received mayalso include, without limitation, data indicating the distance of onevehicle from another vehicle and the angle or trajectory of a vehicle.The process then calculates an estimated position based on theinformation received (step 1506), with the process terminatingthereafter.

Although this process has been illustrated with respect to obtaininglocalization information for the vehicle, the process may be applied toobtain other information for localizing other objects. For example,localization information may be requested and received to identifyobjects around the vehicle. In this manner the vehicle may identifyobstacles.

With reference now to FIG. 16, a flowchart illustrating a process fortransmitting localization information to another vehicle is depicted inaccordance with an illustrative embodiment. The process may beimplemented by machine controller 302 utilizing communications unit 312in FIG. 3.

The process begins by receiving a vehicle localization informationrequest (step 1602) from another vehicle. For example, a vehicle, suchas combine/harvester 106, which has lost sensor capabilities necessaryfor determining a vehicle position estimate, may request informationfrom a sensor system of another vehicle working in the same worksite.Next, the process performs vehicle localization (step 1604), using thesensor system of the vehicle to determine a position estimate of thevehicle in relation to a map or route. Then, the process sends thevehicle localization information to the vehicle that sent the request(step 1606), with the process terminating thereafter.

With reference now to FIG. 17, a flowchart illustrating a process forselecting sensor data to be used is depicted in accordance with anillustrative embodiment. This process may be implemented by a softwarecomponent, such as sensor processing algorithms 604 or machine controlprocess 602 in FIG. 6.

The process begins by receiving sensor data (step 1702) from a pluralityof sensors in a sensor system, such as sensor system 500 in FIG. 5,located on a vehicle, such as one of combine/harvesters 104, 106, and108. The process determines whether all sensor data has been received(step 1704) from the sensor system. If all of the sensor data has notbeen received, the process returns to step 1702 and receives furthersensor data. If all of the sensor data has been received, the processthen determines an accuracy level for the sensor data (step 1706) ofeach sensor from which sensor data was received. For example, if sensordata was received from four sensors, a global positioning system, avisible light camera, a lidar, and an infrared sensor, the accuracylevel of the sensor data from each sensor is determined. Analyzing thesensor data for accuracy involves determining an accuracy level for thesensor data based on the sensor data relative to other sensor data andthe confidence level in the sensor. For example, if a sensor is out of amargin of error for distance, angle, position, and the like, it islikely that the sensor has failed or is compromised and should beremoved from the current calculation. Repeated excessive errors aregrounds for declaring the sensor failed until a common mode root causeis eliminated, or until the sensor is repaired or replaced. Once theaccuracy level is determined, the process selects sensor data to be used(step 1708), with the process terminating thereafter.

With reference now to FIG. 18, a flowchart illustrating a process forsensor data fusion is depicted in accordance with an illustrativeembodiment. This process may be implemented by a software component,such as sensor processing algorithms 604 or machine control process 602in FIG. 6.

The process begins by receiving sensor data (step 1802) from a pluralityof sensors in a sensor system, such as sensor system 500 in FIG. 5,located on a vehicle, such as one of combine/harvesters 104, 106, and108. The process determines an accuracy level for the sensor data (step1804) and selects sensor data to fuse (step 1806). The accuracy levelwill influence which sensor data is fused and which sensor data isconsidered an outlier. The process then fuses the selected sensor datato form a single value (step 1808), with the process terminatingthereafter. Sensor data is fused by mathematically processing the sensordata to obtain a single value used to determine relative position. Thesingle value may then be shared with multiple vehicles. For example, ifa vehicle experiences sensor component failure and requests sensor datafrom another vehicle, the single value may be shared to aid the vehiclewith sensor component failure in determining the relative position ofthe vehicle.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a,” “an,” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

The description of the present invention has been presented for purposesof illustration and description, and is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the art. Theembodiment was chosen and described in order to best explain theprinciples of the invention, the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

1. A method for managing sensor data for a vehicle, the methodcomprising: attempting to receive sensor data at a vehicle having aplurality of sensors; responsive to an inability of the vehicle toobtain needed sensor data, requesting collected sensor data from aplurality of other vehicles to form alternate sensor data; and receivingthe alternate sensor data and controlling the vehicle using thealternate sensor data.
 2. The method of claim 1, wherein the controllingstep comprises: performing localization of the vehicle using thealternate sensor data to identify a location of the vehicle.
 3. Themethod of claim 1, wherein the controlling step further comprises:controlling movement of the vehicle based on the location of thevehicle.
 4. The method of claim 1, wherein the plurality of sensors is aplurality of redundant sensors.
 5. The method of claim 1, wherein thevehicle has a set of fields of regard covered by the plurality ofsensors.
 6. The method of claim 5, wherein an absence of an expectedfield of regard within the set of fields of regard for the vehicleresults in an absence of the needed sensor data.
 7. The method of claim6, wherein the absence of the expected field of regard is caused by oneof a sensor failure within the plurality of sensors and an environmentaround the vehicle.
 8. A method for managing sensor data for a pluralityof vehicles, the method comprising: receiving a request, at a firstvehicle, for sensor data from a different vehicle; collecting the sensordata from a plurality of sensors at the first vehicle; and sending thesensor data to the different vehicle.